1. Field of the Invention
This invention relates in general to data buffering, switching and routing in computer networks, and in particular to buffering, switching and routing data between devices or interfaces having disparate data protocols and data formats.
2. Description of the Related Art
Computer networks consist of various elements that process data, store data, or transfer data between the elements. Typically, each of the network elements manages the data in disparate forms. For example, an operating system running on a processor element typically deals with data based upon the address of the data stored in a local memory. The data is transferred between the processor and memory on a processor or memory bus using one type of data transfer protocol. Similarly, data may be transferred between the processor or the memory and an I/O device in a system on an I/O bus, such as a PCI bus, using a different data protocol. Still further, data may be transferred in packets between two different computers on a network cable, such as an Ethernet cable, using a third protocol.
It is necessary for elements within a network desiring to transfer data between one another to convert or translate from one data protocol to another when moving the data between the elements. Consider, for example, a network interface card (NIC) that transfers data between an Ethernet cable and a PCI local I/O bus. The NIC includes an Ethernet interface, which is an example of a packetized data interface, for receiving a packet of data one bit at a time according to the Ethernet serial cable protocol. The NIC also includes a PCI interface, which is an example of an addressed data interface, for transferring the data packet to system memory according to the PCI bus protocol. Conversely, the NIC is capable of receiving a packet of data from system memory on the PCI bus and transmitting the data on the Ethernet cable.
For at least two reasons, the NIC may also be required to buffer the data in a memory on the NIC in between receiving the data from one interface and transmitting the data on the other interface. First, the PCI bus protocol and Ethernet protocol have different data transfer rates. In particular, the PCI bus is capable of transferring data at a much faster rate than the Ethernet protocol. Hence, if the NIC does not buffer the data from the PCI interface, some of the data may be lost before it can be transmitted on the Ethernet interface.
Second, the two protocols have different arbitration techniques. In the PCI protocol, a device desiring to transfer data must first arbitrate for and be granted ownership of the bus at which time it may transfer up to a maximum amount of data before relinquishing ownership of the bus. In contrast, Ethernet devices may transmit at will; however, they are subject to collision back off and retry algorithms. Hence, if the NIC does not buffer the data from the Ethernet interface, some of the data may be lost before the PCI interface can arbitrate for and be awarded ownership of the PCI bus, or vice versa if a collision occurs and the time to successfully retry is relatively lengthy.
Recently, a new packetized data transfer protocol, the InfiniBand™ Architecture (IBA), has been defined for performing data transfers between computers and their peripheral devices, such as storage devices and network interface devices, and between computers themselves. The IBA is a high speed, highly reliable, serial computer interconnect technology. The IBA specifies interconnection speeds of 2.5 Gbps (Gigabits per second) (1× mode), 10 Gbps (4× mode) and 30 Gbps (12× mode) between IB-capable computers and I/O units.
The IBA specifies InfiniBand (IB) channel adapters (CA) for adapting an InfiniBand packetized data link to another data transfer protocol, such as an addressed data bus like the PCI bus. Channel adapters enable a host computer or I/O device to connect to an IBA network by performing protocol translation between the IBA protocol and the local bus protocol, such as PCI. For example, in response to programming from a local processor, the CA may transfer a payload of data from a memory on the local bus, create an IB packet with the data, and transmit the packet to another IB node. Conversely, the CA may receive an IB packet, extract the payload of data from it, determine where in the memory the payload belongs, and transfer the payload data on the local bus to the memory. This is a protocol translation function, commonly referred to as a transport layer function, performed by the CA in addition to the protocol translation performed by the NIC described above, which simply transfers the entire packet from one interface to another, from the addressed data interface to the packetized interface and vice versa. In the case of the NIC, the local processor performs the transport layer function.
In addition to channel adapters, the IBA also specifies IB switches, for switching IBA packets within a local subnet of IBA channel adapters. IB switches include multiple IB ports for connecting to multiple IB channel adapters on IB links. Each channel adapter port has a unique local subnet identification number. The IB switches maintain switching tables that enable the switches to receive a packet in one port and switch the packet out another port that is connected to the destination channel adapter or to another switch from which the destination channel adapter is downstream.
It is desirable to include the functionality of an IB channel adapter and switch together in a single device, which may be referred to as a hybrid channel adapter/switch. A hybrid channel adapter/switch would be required to perform two functions. First, it would have to perform protocol translation between an IB link and a local bus, such as PCI. Second, it would have to perform IB packet switching between the various IB ports.
It is also highly desirable for the hybrid channel adapter/switch to perform a third function: addressed data switching or bridging. This function derives from the fact that it is desirable to have multiple addressed data local bus interfaces or ports, such as multiple PCI bus interfaces.
Multiple local bus ports are necessary to supply sufficient data throughput to match the IB port bandwidth, due to the enormous data throughput IB links are capable of sustaining. Furthermore, local bus loading specifications, such as PCI, limit the number of I/O controllers that may be attached to a local bus. Hence, it is also desirable to have multiple local bus ports on the hybrid channel adapter/switch to enable more I/O controllers to be connected than a single local bus can support. In such an arrangement, the I/O controllers on disparate local buses need to be able to transfer addressed data to one another. Additionally, an I/O controller will need to transfer data to or from a system or local memory that is connected to a memory controller on a different local bus than the I/O controller. Hence, the hybrid channel adapter/switch must be capable of switching or bridging addressed data between its multiple local bus interfaces.
For similar reasons described above with respect to the NIC, the hybrid channel adapter/switch would also need to perform data buffering, not only for the protocol translation function, but also for the packet switching and addressed data bridging functions. One obvious solution is to extend the architecture of the NIC described above. That is, the hybrid channel adapter/switch could provide a dedicated buffer memory for each possible data transfer path between each of the interfaces in the hybrid channel adapter/switch. A buffer between each of the combinations of paths between the IB interfaces would provide packet buffering for the packet switching function. Similarly, buffers between the PCI interfaces would provide addressed data buffering for the local bus bridging function. Buffers between the protocol translation device and the IB and PCI interfaces would provide buffering for the protocol translation function.
However, there are drawbacks to the solution just described. First, the solution makes inefficient use of the buffering memories. Due to the high transfer rates of the interfaces in question, the size of the buffers on many of the paths may have to be relatively large in order to achieve optimum transfer rates in the worst-case conditions. Hence, the solution may require more total buffer memory than is realistically manufacturable given contemporary semiconductor manufacturing capabilities. Consequently, a decision must be made a priori concerning how the limited buffer memory space will be statically divided among the data paths between the various interfaces.
This static allocation may at times result in inefficient use of the buffering resources. For example, the flow of data traffic through the hybrid channel adapter/switch may be such that for some periods some of the data paths are not used and others are saturated. In this situation, the buffers in the idle data paths are sitting empty. In contrast, because the heavily used paths do not have sufficient buffering resources, the overall data throughput rate through the hybrid channel adapter/switch may be reduced. A more efficient solution would enable the unused buffers to be used to facilitate the flow of data through the paths needing additional buffering resources.
A second drawback is that the solution requires double-buffering in the case of the transport layer protocol translation function. For example, a packet is received by an IB interface and stored in a buffer on the data path between the IB interface and the transport layer device. The transport layer device then reads the packet from the buffer, performs the necessary translation function, and writes the translated data to a buffer on the data path between the transport layer device and the destination local bus interface. The local bus interface then reads the translated data from the second buffer and transfers the data on the local bus. This double-buffering, i.e., buffering the data twice within the hybrid channel adapter/switch, works to reduce the overall bandwidth of data that may be transferred through the hybrid channel adapter/switch.
Therefore, what is needed is an apparatus in a network adapter having multiple disparate data protocol interfaces or devices to transfer data both between interfaces of the same type and between interfaces of a different type. The apparatus needs to perform the transfers without double-buffering the data and in such a manner that makes efficient use of valuable, limited buffer memory resources.